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EXECUTIVE  SUMMARY 


This  report  describes  processing  of  dynamic  and  cued  MetalMapper  data  collected  as  part  of  the 
Environmental  Science  Technology  Certification  Program  (ESTCP)  demonstration  conducted  in 
2015  at  the  former  Port  Ord,  CA.  The  data  was  processed  under  project  ESTCP  MR-  201421 
“Practical  UXO  Classification:  Enhanced  Data  Processing  Strategies  for  Technology  Transition.” 
The  objective  of  this  project  is  to  demonstrate  advanced  methods  for  the  detection  and 
classification  of  unexploded  ordnance  (UXO)  in  a  production  setting.  At  the  Port  Ord 
demonstration,  the  specific  classification  objectives  were: 

1.  Demonstrate  if  large  munitions,  such  as  155mm  projectiles  to  a  depth  of  2  feet,  can  be 
confidently  classified  within  the  range  of  background  conditions  (medium  to  high 
metallic  density)  that  exist  in  the  Impact  Area  at  Port  Ord. 

2.  Demonstrate  if  smaller  munitions  such  as  40mm  projectiles  can  be  confidently  classified 
within  the  range  of  background  conditions  at  Port  Ord. 

3.  Classify  all  TOT 

A  detailed  evaluation  of  the  performance  objectives  outlined  in  the  demonstration  plan  is 
provided  in  a  companion  report  under  ESTCP  project  MR-201420.  Here  we  present  the 
following  key  results  from  the  processing  of  dynamic  and  cued  data: 

1.  Target  picking  at  a  late  time  channel  (e.g.  9.2  ms)  in  the  dynamic  MetalMapper  data 
significantly  reduces  the  amount  of  clutter  detected.  Earge,  slow  decaying  targets  that  are 
underneath  clutter  and  are  undetectable  in  EM61  data  can  be  resolved  at  late  times  in  the 
MetalMapper  detection  data.  We  recommend  that  long  off-time  dynamic  MetalMapper 
data  is  used  for  detection  of  large  TOI  in  follow-on  work  at  Port  Ord. 

2.  The  primary  objective  of  confident  classification  of  large  munitions  such  as  155mm 
projectiles  was  achieved  using  the  MetalMapper  cued  data  acquired  in  low,  medium  and 
high  density  grids  at  Port  Ord. 

3.  Although,  there  were  11  small  TOI  missed  in  our  cued  MetalMapper  diglist,  the 
classification  results  indicate  that  95%  of  smaller  munitions  can  be  confidently  classified 
within  the  range  of  background  conditions  at  Port  Ord. 
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1.0  INTRODUCTION 


The  former  Fort  Ord,  located  in  Monterey  County,  California,  is  a  closed  installation  that  has  an 
ongoing  munitions  response  program  managed  by  the  U.S.  Corps  of  Engineers  (USACE) 
Sacramento  District  on  behalf  of  the  Eort  Ord  Base  Realignment  and  Closure  (BRAC)  Office.  In 
2014,  ESTCP  conducted  a  demonstration  of  advanced  classification  technology  at  Eort  Ord.  The 
objectives  of  this  project  were  to  demonstrate  the  ability  to  collect  data  in  dynamic  and  cued 
modes  for  munitions  and  explosives  of  concern  (MEC)  detection  and  identification  using  the 
MetalMapper,  while  simultaneously  transferring  the  technology  from  the  researchers  to 
production  companies  and  to  gain  regulator  acceptance. 

The  subject  area  was  within  Units  11  and  12  in  the  Impact  Area  MRA  at  the  former  Eort  Ord. 
Units  II  and  12  cover  approximately  476  acres.  The  complete  area  had  full  coverage  surface 
MEC  remediation,  plus  digital  geophysical  mapping  (DGM)  was  performed  using  a  vehicle 
towed  array  of  three  Time-domain  Electromagnetic  (TDEM)  metal  detectors  (EM61-MK2)  over 
the  complete  site  except  small  areas  where  terrain  and  a  stand  of  oak  trees  precluded  data 
collection  (Eigure  1) 
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Figure  1.  Units  11  and  12  of  the  Ft.  Ord  Impact  Area. 

Gridded  image  shows  EM-61  detection  map.  MetalMapper  grids  surveyed  under  this  demonstration 

project  are  identified  by  blue  polygons. 
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Surface  sweeps  identified  MEC  items  throughout  Units  11  and  12,  including  37mm,  40mm, 
57mm,  60mm,  75mm,  90mm,  105mm,  and  155mm  projectiles.  The  results  of  the  DGM 
following  surface  removal  showed  very  high  EM61-MK2  anomaly  densities  throughout  most  of 
the  two  Units.  The  results  of  the  DGM  indicated  that  a  variety  of  MEC  items  may  be  present  in 
the  subsurface.  The  BRAG  Office  was  particularly  interested  in  determining  if  advanced 
classification  methods  could  be  used  to  locate  large  MEC  items  such  as  155mm  projectiles,  as 
well  as  sensitive  munitions  such  as  40mm  projectiles. 

Given  that  there  was  a  wide  variety  of  possible  munitions  and  that  the  area  was  considered  to 
have  very  high  anomaly  density,  a  primary  goal  was  to  demonstrate  if  large  munitions  such  as 
155mm  projectiles  at  depths  to  2  feet  could  be  confidently  classified  within  a  challenging  high 
metallic  debris  background.  In  order  to  determine  if  this  was  possible,  four  representative  grids 
of  high,  medium,  and  low  anomaly  densities  (Eigure  1)  were  surveyed  with  the  MetalMapper  in 
both  dynamic  and  cued  modes  for  detection  and  classification,  respectively. 

In  the  remainder  of  this  report  we  provide  a  brief  description  of  the  hardware  and  data 
processing  technologies  that  were  used  for  this  demonstration.  We  discuss  detection  of  TOI 
using  dynamic  MetalMapper  data,  and  then  in  section  4.0  turn  to  classification  with  cued 
MetalMapper  data.  We  provide  a  summary  of  our  classification  approach  and  a  retrospective 
analysis  for  small  TOI  that  were  missed  in  our  classification  diglist.  We  conclude  with  lessons 
learned  and  recommendations  for  future  work. 
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2.0 


TECHNOLOGY  DESCRIPTION 


2.1  DETECTION 

The  MetalMapper  is  an  advanced  time-domain  electromagnetic  (TDEM)  sensor  designed 
specifically  for  classification  of  MEC.  The  antenna  platform  includes  three  mutually  orthogonal 
transmitter  loops  and  a  spatial  array  of  seven  3-axis  receiver  antennas  providing  21  independent 
measurements  of  the  transient  secondary  magnetic  field. 


Figure  2.  Dynamic  MetalMapper  survey  at  Ft.  Ord. 

Advanced  TDEM  sensors  such  as  the  MetalMapper  acquire  detection  data  for  digital  geophysical 
mapping  (DGM)  with  a  higher  resolution  and  longer  off-times  than  conventional  EM-61  arrays 
(Eigure  3).  These  features  are  particularly  advantageous  at  a  highly  cluttered  site  such  as  Eort 
Ord  where  the  primary  target  of  interest  is  a  large,  slow  decaying  munition. 
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Figure  3.  Comparison  of  EM61  (left)  and  MetalMapper  (right)  Sensors. 

Within  each  panel  we  show  an  image  of  the  sensor  (top  left),  sensor  geometry  (top  right,  transmitters  in 
red,  receivers  in  black)  and  measurement  off-times  (bottom).  For  the  MetalMapper,  three  possible  off- 

times  are  shown,  extending  as  late  as  22.5  ms. 
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2.2  CLASSIFICATION 


Target  classification  is  usually  carried  out  using  cued  interrogation  data  acquired  over  anomalies 
initially  identified  in  the  detection  data.  These  cued  interrogations  eliminate  relative  positional 
errors  by  acquiring  data  with  a  stationary  sensor.  The  multi-static,  multi-component  geometry  of 
advanced  sensors  such  as  the  MetalMapper  allows  for  reliable  target  characterization  with  a 
single  cued  sounding.  In-field  inversions  of  cued  soundings  help  to  ensure  that  the  sensor  is 
optimally  positioned  over  each  target. 

Cued  interrogation  data  are  inverted  using  a  dipole  model  to  recover  estimates  of  extrinsic 
(location,  depth,  and  orientation)  and  intrinsic  (dipole  polarizabilities)  parameters  for  each 
interrogated  target.  The  estimated  polarizabilities  for  each  recovered  dipole  source  are  then 
matched  against  a  pre-defined  library  to  identify  likely  targets  of  interest  (TOI)  at  the  site.  For 
this  demonstration,  all  classification  processing  was  carried  out  using  the  UXOLab  software 
package  developed  jointly  by  BTG  and  UBC-GIF. 
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3.0  DETECTION  WITH  THE  METALMAPPER  AT  EORT  ORD 


For  this  demonstration  we  investigated  whether  dynamic  MetalMapper  measurements  could  identify 
responses  from  large  TOI  at  late  times,  after  small  metalhc  surface  debris  has  decayed  away. 

Extending  the  dynamic  MetalMapper  off-time  window  out  to  approximately  25  ms  while 
maintaining  an  along-track  sampling  rate  of  0.1s  necessarily  reduced  the  number  of  stacks  for 
each  dynamic  sounding.  This  reduced  the  signal  to  noise  ratio  (SNR)  of  the  dynamic  data.  To 
ensure  that  noise  levels  did  not  preclude  detection  of  large  TOI,  we  collected  dynamic 
MetalMapper  data  at  the  testpit  over  a  range  of  items.  We  found  that  we  could  both  detect  and 
recover  high  quality  polarizabilities  for  a  range  of  TOI  using  dynamic  testpit  data.  In  order  to 
ensure  that  these  testpit  findings  held  for  the  high  metallic  debris  grids  at  the  site,  we  also  went 
out  into  the  high  density  1 1-B  grid  and  collected  a  swath  of  dynamic  data  with  each  of  the  short, 
medium  and  long  time  windows.  A  quality  control  (QC)  seed  was  detected  in  the  surveyed 
swath,  and  our  measurements  confirmed  that  both  detection  and  recovery  of  high  quality 
polarizabilities  were  feasible  with  a  long  off-time  in  this  survey  environment. 

Figure  4  compares  the  unfiltered  data  for  the  latest  time  channel  (1.26  ms)  of  EM61  data  acquired  at 
Fort  Ord  grid  1 1-B  with  a  comparable  time  channel  (1.23  ms)  of  dynamic  MetalMapper  data.  Earge, 
distinct  anomalies  are  present  in  both  data  sets  and  correlate  well  with  each  other.  As  expected,  the 
dynamic  MetalMapper  does  a  better  job  of  resolving  individual  targets  in  the  dense  region  in  the 
northeast  comer  of  the  grid.  MetalMapper  dynamic  data  are  more  susceptible  to  small  near  surface 
metal  as  its  receivers  are  physically  closer  to  the  surface.  However,  as  shown  in  Figure  5  the 
MetalMapper’ s  late  time  measurements  attenuate  the  responses  of  small  scrap,  while  responses 
from  larger,  slow-decaying  targets  remain  detectable.  The  late  time  MetalMapper  detection  data  is 
therefore  ideally  suited  to  the  primary  objective  of  detecting  155  mm  projectiles.  Figure  6  shows 
polarizabilities  extracted  from  four  likely  TOI  targets  identified  in  the  late-time  detection  data.  All 
of  these  items  are  a  good  match  to  large  targets  of  interest  in  the  ordnance  library. 


60  m 


Figure  4.  Left:  EM61  Detection  Data  in  Ft.  Ord  grid  11-B  at  1.26  ms.  Right:  Dynamic 
MetalMapper  Data  at  1.24  ms,  Collected  Over  the  Same  Area. 
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60  m 


Figure  5.  MetalMapper  Detection  Data  in  Grid  11-B  at  0.63  ms  (left),  1.23  ms 

(middle),  9.2  ms  (right). 

The  northeast  corner  appears  highly  cluttered  at  early  times  but  reveals  distinct  individual 

targets  at  later  times. 


Figure  6.  Left:  MetalMapper  Detection  Data  in  Grid  11-B  at  9.2  ms.  White  Circles  and 
Numbers  Indicate  the  Target  Picks  shown  on  right.  Right:  Estimated  Polarizabilities  (solid 
lines)  Recovered  from  Cued  Inversions  for  the  Indicated  Targets.  Dashed  Lines  are  the 

Best  Matching  Library  Polarizabilities. 


We  picked  targets  in  the  MetalMapper  dynamic  data  using  vertical  component  profiles  in  all 
seven  receivers.  A  separate  worst-case  detection  threshold  was  calculated  for  each  receiver,  and 
initial  picks  from  individual  profiles  were  subsequently  merged  using  a  minimum  anomaly 
footprint  determined  using  the  UXOLab  detection  modeler.  Table  1  summarizes  the  total 
number  of  picks  in  each  of  the  four  MetalMapper  grids. 
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Table  1. 


Summary  of  Target  Picks  in  MetalMapper  Grids. 


Area 

Density  of  metallic 
debris 

Target  picks 

Average  target  picks 
per  too  ft  subgrid 

12A 

Low 

1698 

283 

12B 

Medium 

1878 

470 

llA 

High 

3646 

608 

11-B 

High 

3173 

793 
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4.0  CUED  METALMAPPER  PROCESSING 


4.1  FEATURE  EXTRACTION 

Cued  data  were  acquired  in  four  different  grids  (Table  2  and  Figure  7)  at  anomalies  picked  from 
dynamic  MetalMapper  data.  In  addition,  cued  data  were  acquired  at  anomalies  picked  from 
EM61  data  in  the  vicinity  of  grids  11-A  and  11-B.  The  total  number  of  targets  for  the  entire 
dataset  is  2803. 


Table  2.  Summary  of  Targets  in  Each  Grid. 

Grid  11-EM61  refers  to  data  acquired  at  anomalies  in  the  vicinity  of  grids  11-A  and  11-B  picked  from 
EM61  data.  Target  densities  are  based  on  number  of  targets  in  the  full  coverage  sub-grids,  where  all 
targets  were  cued.  “Targets  other”  refers  to  potential  TOl  targets  that  were  cued  in  all  other  sub-grids. 


Grid 

Density 

Total 

targets 

Targets  full 
coverage 

Targets 

other 

Target  density 
(per  acre) 

12-A 

Low 

403 

272 

131 

1186 

12-B 

Medium 

494 

417 

77 

1813 

11-A 

High 

823 

653 

170 

2831 

11-B 

Highest 

934 

790 

144 

3428 

11-EM61 

High 

150 

NA 

150 

NA 

4.0513 
4  0512 
40512 
4.0511 
40511 
40511 
4.051 
4.051 
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Figure  7.  Relative  Location  of  the  Four  Grids  in  which  MetalMapper  Data  Were 

Acquired  at  Fort  Ord. 

Blue  lines  are  outlines  dividing  each  grid  into  either  four  or  six  100x100 foot  sub-grids.  Black  dots 
indicate  sub-grids  in  which  all  anomalies  were  cued.  Within  the  other  sixteen  sub-grids  only  anomalies 
previously  identified  from  inversion  of  dynamic  MetalMapper  data  as  potential  TOI  were  cued. 


MetalMapper  cued  data  for  all  anomalies  were  received  as  a  set  of  TEM  files  for  the  cued 
anomaly  data,  background  measurements  and  IVS  and  test  pit  measurements.  TEM  files  were 
converted  to  CSV  using  TEMPlot  (EMSDPlot).  CSV  files  were  imported  into  UXOLab. 
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On  import  UXOLab  automatically  performs  background  corrections,  using  the  background  that 
was  collected  closest  in  time  for  each  anomaly.  The  data  were  inverted  in  UXOLab  using  a 
sequential  inversion  approach  to  estimate  target  location,  depth  and  primary  polarizabilities. 
Instrument  height  above  the  ground  was  assumed  to  be  12  cm.  Noise  standard  deviation 
estimates  were  based  on  more  than  80  background  measurements.  Target  location  was 
constrained  to  lie  between  +0.75  m  in  both  X  and  Y  directions  relative  to  the  acquisition 
location.  Target  depth  was  constrained  to  lie  between  -1.3  and  0  m.  The  initial  optimization  for 
target  location  identified  up  to  eight  starting  models  to  input  into  the  subsequent  estimation  of 
polarizabilities.  We  performed  three  inversions  per  anomaly,  solving  for  (1)  a  single  object 
(single  object  inversion:  SOI);  (2)  two  objects  (201);  and  (3)  three  objects  (301). 

Analysis  of  the  data,  including  visual  QC  of  data  and  model  parameters,  selection  of  training 
data,  and  dig  list  creation,  was  performed  using  the  UXOLab  software  suite.  Visual  QC  of  the 
data  was  performed  using  the  UXOLab  module  QCZilla,  which  provides  a  thorough  overview  of 
the  observed  and  predicted  data,  predicted  model  parameters,  and  measures  of  data/model 
quality.  Predicted  polarizabilities  were  compared  to  reference  polarizabilities  for  various 
ordnance  items  initially  derived  from  IVS  and  test  pit  measurements.  The  Fort  Ord  IVS 
comprised  only  three  items:  small,  medium  and  large  ISOs.  Test  pit  measurements  were  made 
over  several  items,  including  40mm,  57mm,  60mm,  75mm,  81mm,  several  variants  of  155mm, 
2.36”  rocket,  Stokes  mortar,  and  ISOs.  As  the  analysis  proceeded,  the  library  of  reference  items 
was  augmented  with  additional  items  based  on  a  comparison  with  TOI  from  previous  live  sites, 
and  from  ground  truth  obtained  through  training  data  requests  and  partial  ground  truth.  Each  item 
in  the  ordnance  reference  library  was  assigned  a  size  (diameter)  in  mm.  Each  item  with  a  dig 
decision  of  “dig”  in  the  submitted  dig  list  was  assigned  a  size  category  (1  for  diameter  <50mm;  2 
for  50<diameter<=  100mm;  and  3  for  diameter>  100mm)  based  on  the  ordnance  item  in  the 
reference  library  with  the  best  matching  primary  polarizability  (El). 

During  data/model  QC  the  primary  objectives  were  to  (1)  flag  high-likelihood  TOI  anomalies; 
(2)  flag  anomalies  to  be  requested  as  training  data;  and  (3)  fail  bad  models  and  inversions. 
Anomalies  flagged  as  high-likelihood  TOI  were  monitored  during  the  dig  list  creation  phase  to 
ensure  they  were  being  dug,  ideally  early  in  the  dig  list.  Models  and  inversions  were  considered 
to  be  bad  when  the  inversion  failed  (i.e.,  the  data  misfits  are  large),  or  when  the  recovered  model 
location(s)  were  on,  or  near,  an  inversion  boundary  (i.e.,  significantly  outside  the  footprint  of  the 
sensor).  Bad  models  and  inversions  were  identified  in  a  semi-automated  manner.  E.g.,  models 
would  be  sorted  by  different  measures  of  polarizability/data  quality  and  the  visual  QC  process 
would  focus  on  the  models  with  the  poorest  quality. 

With  multi-object  inversions  it  is  not  uncommon  that  one  of  the  models  is  unrealistic  (e.g., 
deep,  large  in  magnitude,  sometimes  located  on  or  near  a  horizontal  inversion  boundary)  yet 
provides  the  best  fit  to  the  reference  polarizabilities  (e.g.,  Eigure  8).  In  such  cases  the  model 
was  flagged  as  failed.  Models  flagged  as  failed  were  not  used  in  the  classification  process. 
Anomalies  with  all  models  from  all  inversions  failed  were  classified  as  “cannot  extract 
reliable  parameters”;  these  anomalies  were  dug.  Eor  the  Eort  Ord  dataset,  ten  anomalies  (EO- 
10493,  25094,  25577,  35250,  35296,  36056,  40023,  40040,  40225  and  40263)  were  classified 
as  “cannot  extract  reliable  parameters”.  An  example  is  shown  in  Eigure  9.  Eor  a  given 
anomaly,  if  more  than  one  model  was  passed  the  classification  procedure  will  consider  all 
passed  models  and  effectively  use  the  one  that  is  “best”  based  on  the  classification  metric. 
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For  anomalies  with  recollects,  the  classification  procedure  will  consider  all  passed  models  from 
all  versions  of  the  anomaly  and  use  the  one  that  is  “best”  based  on  the  classification  metric. 


T30499-C2434.SOH-P 


T30499-C2434-20I/1-P  T30499-C2434-20l'2-F 


Target  location 

x=0.75  Y=-0.75  Target  depth  Z=1 . 30 


Figure  8.  Example  of  an  Unrealistic  Model  (FO-30499;  frag). 

Top  row  shows  models  from  SOI  and  201;  middle  row  shows  models  from  301.  For  models  6  (box  with 
broken  red  lines)  the  predicted  depth  is  1.3m  and  the  predicted  horizontal  location  is  at  a  corner  of  the 
inversion  boundaries.  The  polarizabilities  for  these  models  are  high  in  amplitude  and  have  a  jittery 
appearance;  classic  signs  that  the  model  is  an  artifact  of  the  multi-object  inversion  process.  Accordingly, 
this  model  was  failed  during  QC.  Lower  left  plot  shows  predicted  model  locations  (number  circles) 
relative  to  the  instrument  receivers  (grey  lines)  and  transmitters  (grey  dots).  Broken  black  line  denotes 
the  horizontal  inversion  bounds.  Plot  at  lower  right  shows  predicted  model  depths  (numbered  circles). 
The  yellow  highlighted  circle  in  both  plots  at  the  bottom  is  model  6. 
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Rx7  Rx6  Rx5  Rx4  Rx3  Rx2  Rxl 


RxX-TxX  RxY-TxX  RxZ-TxX  RxX-TxY  RxY-TxY  RxZ-TxY 


T40023-C2843-SOI/1-F 


T40023-C2843-20n-F 


T40023-C2843-2Ol^2-F 


Figure  9.  Example  of  an  Anomaly  with  No  Passed  Models  (“cannot  extract  reliable 

parameters”):  FO-40023;  large  frag). 

Top:  data  fits  (blue  —  observed;  green— predicted).  Bottom:  polarizabilities.  The  data  fits  for  all 

inversions  were  very  poor. 
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The  Fort  Ord  MetalMapper  Cued  dataset  comprised  2803  unique  anomalies.  Of  the  21696  total 
models  (including  models  from  recollects),  13107  were  passed  and  used  in  the  classification 
process;  8589  were  failed.  Initially,  446  anomalies  were  classified  as  “high  likelihood  UXO” 
during  QC,  though  some  of  these  correspond  to  shared  anomalies.  Of  the  446,  about  375 
correspond  to  unique  locations.  370  of  these  (83%)  correspond  to  TOI,  though  several  of  these 
are  shared  targets.  The  total  number  of  unique  TOI  in  the  MetalMapper  Cued  dataset,  excluding 
shared  targets,  is  356. 

4.2  CLASSIFICATION 

4.2.1  Training  data  selection 

Figure  10  shows  the  distribution  of  models  in  decay  versus  size  feature  space.  The  overlap 
between  the  majority  of  features  and  reference  items  suggests  that  this  may  be  a  challenging  site 
for  classification  because  a  relatively  large  proportion  of  the  non-TOI  will  have  size/decay 
characteristics  similar  to  the  TOI.  Our  ad  hoc  dataset  degree  of  difficulty  measure,  which  is 
based  on  various  polarizability  and  dataset  metrics,  for  this  site  suggests  it  will  be  significantly 
more  difficult  than  previous  ESTCP  demonstrations  at  MMR  and  Beale,  but  easier  than  Rucker 
and  New  Boston. 
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Figure  10.  Distribution  of  Models  in  Decay(t75,tl)  versus  Size(tl)  Feature  Space^where 
size(tl)  is  the  total  polarizability  measured  at  the  first  time  channel  (tI=0.II7  ms),  and 
decay(t75,tl)  is  size(t75)/size(tl)  where  t75=5.232  ms.  Some  outliers  are  not  shown.  Stars 
represent  selected  ordnance  library  reference  items  ranging  in  size  from  37mm  to  1 55mm. 
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Our  classification  method  is  based  on  polarizability  matching  with  respect  to  ordnance  items  in  a 
reference  library.  For  this  approach  to  be  successful  it  is  important  to  determine  the  types  of 
ordnance  present  at  the  site.  During  visual  QC  the  analyst  keeps  track  of  suspicious,  UXO-like 
items  (i.e.,  items  with  modeled  polarizabilities  possessing  UXO-like  properties).  Training  data 
for  some  of  these,  particularly  those  with  polarizabilities  that  are  somewhat  different  from  the 
items  in  the  reference  library,  would  be  requested.  In  addition,  we  used  our  custom  training  data 
selection  tool,  TrainZilla,  to  explore  feature  space  and  automatically  search  for  clusters  of  items 
with  self-similar  polarizabilities.  In  TrainZilla,  the  user  selects  a  region  in  feature  space  by 
drawing  a  polygon,  and  the  program  automatically  identify  clusters  of  self-similar  feature  vectors 
by  computing  a  misfit  matrix  M  with  elements 

where  is  the  log-transformed  total  polarizability  for  the  feature  vector.  Feature  vectors 
with  mutual  misfit  less  than  a  user-specified  threshold  define  a  cluster  in  polarizability  space. 
This  analysis  is  helpful  for  identifying  clusters  that  may  not  be  readily  evident  in  decay-size 
feature  space:  e.g.,  targets  with  consistent  polarizabilities  that  may  be  hidden  in  the  “cloud”  of 
non-TOI  features.  A  basic  example  of  the  use  of  TrainZilla  is  shown  in  Figure  1 1  and  Figure  12. 
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Figure  11.  Example  of  Use  of  the  Training  Data  Selection  Tool  {TrainZilla). 

A  polygon  (solid  black  line)  is  drawn  in  feature  space.  Clusters  of  items  with  self-similar  polarizabilities 
are  automatically  found  based  on  the  specified  cluster  search  parameters.  In  this  case  a  cluster 
comprising  13  features  is  visible  (solid  feature  symbols  encompassed  by  broken  line).  Polarizabilities  for 

the  models  in  this  cluster  are  shown  in  Figure  12. 
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Figure  12.  Polarizabilities  for  the  Models  in  the  Cluster  Shown  in  Figure  11. 

Red,  black  and  magenta  lines  are  predicted  polarizabilities.  Broken  grey  lines  are  best  fitting  reference 
polarizabilities.  Target  labels  are  number  following  “t”  in  the  bottom  label  of  each  plot.  Training  data 
were  requested  for  the  5th  item  (magenta-highlighted  index  number).  This  was  confirmed  to  be  a  37mm 
projectile  (armor  piercing  tracer,  M74).  Subsequent  ground  truth  information  showed  that  all  of  these 

items  are  37mm  projectiles. 

Our  training  data  requests  typieally  focus  on:  (1)  items  whose  polarizabilities  exhibit  UXO-like 
properties  distinct  from  those  of  items  in  our  reference  library;  (2)  items  with  polarizabilities 
similar  to  items  in  our  reference  library,  but  with  degraded  quality;  (3)  items  from  a  cluster  that  do 
not  necessarily  have  UXO-like  properties  but  are  from  an  unknown  source;  and  (4)  one-off  items. 

An  alternative  approach  to  selecting  training  data,  and  more  one  suited  to  finding  potential  one-off 
items,  is  to  look  for  items  with  polarizabilities  that  closely  match  items  in  a  large  ordnance  library. 
We  do  this  with  the  UXOLab  module  called  the  Ordnance  Museum  (Figure  13),  which,  for 
MetalMapper  cued  data,  comprises  polarizabilities  for  approximately  470  items  (ranging  in  size 
from  20mm  to  155mm)  from  past  ESTCP  live  site  demonstrations.  Approximately  240  of  these 
items  were  derived  from  the  UX- Analyze  polarizability  library.  With  the  Ordnance  Museum  we 
can  easily  search  for  models  in  our  dataset  with  polarizabilities  similar  to  any  of  the  museum  items. 
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For  Fort  Ord  we  found  several  items  with  close  matches  (polarizability  misfit  <  0.3  calculated 
using  all  three  polarizabilities)  to  Ordnance  Museum  items.  Four  representative  items  are  shown  in 
Figure  14.  We  requested  training  data  for  all  of  these  items;  one  of  these  was  a  TOI. 


HI  Figure  1501:  Ordnance  Museum  table  (MetalMapper  Cued:  473  items) 


Ordnance  Museum  ^ 


Figure  13.  UXOLab  Ordnance  Museum  Interface. 


This  is  a  library  of  reference  polarizabilities  compiled  from  several  ESTCP  live  site  demonstrations,  and 
other  projects.  The  ordnance  museum  for  MetalMapper  Cued  data  currently  comprises  approximately 
470  items  ranging  in  size  from  20mm  to  155mm  projectiles. 


16 


T25073-C1722-2OI/1-P 


T25703-C1808-SOI/1-P 


T26284-C1820-SOI/1-P 


T40041-C2857-2OI/2-P 


0.5  1  5  10  0.5  1  5  10  0.5  1  5  10  0.5  1  5  10 


Figure  14.  Polarizabilities  of  Four  Models  with  Close  Matches  to  Items  in  the  Ordnance 

Museum. 

Anomaly  labels  are  the  number  following  the  “T”  in  each  plot  title.  Label  at  top  right  of  each  plot  is  best 
fitting  ordnance  museum  item  name.  Label  at  bottom  left  in  each  plot  shows  the  polarizability  misfit  and 
source  of  the  model.  'We  requested  training  data  for  all  of  these  items.  The  first  was  a  TOI  (75mm);  the 
rest  were  non-TOI.  The  second  item  (FO-25703)  was  an  M48  series  fuze  (non-TOI). 


We  submitted  three  training  requests  prior  to  submitting  our  first  dig  list  for  a  total  of  62  training 
items.  Twenty-eight  of  these  were  TOI  (37mm,  40mm,  57mm,  75mm,  81mm  and  155mm). 
Based  on  our  training  data  we  removed  a  number  of  items  from  our  working  reference  library, 
e.g.,  small  ISO,  Stokes  Mortar,  rockets  (2.36”  and  3.5”)  and  fuzes. 

4.2.2  Classification  method 

Dig  lists  were  developed  using  our  visual  classification  software  DigZilla  (Figure  22,  which  is 
fully  integrated  with  other  elements  of  the  UXOLab  software  suite.  DigZilla  allows  for  the 
creation  of  multi-stage  dig  lists  with  minima!  effort,  and  supports  a  number  of  classifiers. 
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Figure  15.  Screen  Shot  of  the  UXOLab  DigZilla  Graphical  User  Interface. 

Features  in  the  decay  versus  size  feature  plot  are  color  coded  according  to  dig  list  order  (red  earliest, 
black  latest).  White  dots  are  training  items.  Stars  are  reference  items.  Inset  polarizabilities  at  top  left  are 

for  the  currently  selected  item  (black  square). 

We  were  informed  by  the  ESTCP  Program  Office  that  the  Fort  Ord  classification  results  would 
be  scored  twice:  (1)  for  the  primary  objective  of  finding  all  large  TOI  (155mm  projectiles  and 
large  ISOs)  to  a  depth  of  60  cm;  and  (2)  detection  of  all  munitions.  Submission  of  two  separate 
dig  lists  was  permitted  but  not  required.  We  chose  to  submit  a  single  two-phase  dig  list.  For  the 
first  phase  we  use  a  polarizability  library  comprising  only  large  items  (155mm  and  large  ISO). 
For  the  second  phase  we  use  our  full  polarizability  library  (including  the  large  items).  For  both 
phases,  the  classification  approach  was  identical,  based  on  polarizability  misfit  (to  the  best  fitting 
library  reference  item  for  each  model)  using  all  three  polarizabilities.  Misfits  were  calculated 
between  the  first  time  channel  (0.12  ms)  and  channels  46  (12.624  ms),  42  (8.284  ms)  and  39 
(6.038  ms),  respectively  for  FI,  F2  and  F3.  The  end  time  channels  for  the  misfits  were 
determined  automatically  based  on  a  measure  of  polarizability  reliability.  The  ordnance  library 
used  for  the  first  phase  of  the  dig  list  (up  to  dig  number  203)  comprised  12  items;  the  full  library 
used  for  the  second  phase  comprised  35  items  (Figure  16).  The  stop  dig  point  for  this  list  was  dig 
number  667. 


18 


155mm-c  155mm#4-bTP  155mm-a  155mm#4-cTP  155mmA  155mm-b 


155mm  M-d  TP 


155mm#2-aTP 


[ 

0.1 

75mm  A 


13 

t-L _ J_J _ 

75mm 


75mm  F 


- L. 

20; 

j.J... 

Med  ISO  26cm 


81  mm  TP 


37mm  B  75mm  TP 


16! 

_ I._l _ 

75mm  E 


- L. 

- *  - . 

22; 

Med  ISO  65cm 


. . . . .b  . 

17; 

81mm  mortar 


_ L  ■ 

.  _  _ 

23; 

; 

60mm  TP 


75mm  B 


57  mm  A 


- K-* - y 

_ L . 

24; 

j.. 

37mm 


4 

10 

10* 

10® 

10* 

•4 

10 


40mm  TP  40mm 


40rrm  D  40mm  B 


40mm  C 


4 

10 

10* 

c 

10 

lo”^ 

10"* 


Figure  16.  Items  in  the  Ordnance  Reference  Library  Used  for  the  Stage  1  Dig  List. 

The  first  12  items  were  used  in  the  first  phase  of  the  dig  list  (up  to  dig  number  203 )  for  finding  large  TOl. 

The  full  library  was  used  for  subsequent  digs. 


The  stage  1  dig  list  missed  two  QC  seeds.  The  partial  ROC  curve  is  shown  in  Figure  17. 
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TOI1or2 

FtOrd  BlackTusk  PolMatch  None  MetalMapper  Custom  s1  v1 


Figure  17.  Partial  ROC  Curve  for  the  Stage  1  Dig  List. 


Because  two  QC  seeds  were  missed,  we  were  required  to  submit  a  Failure  Analysis  Memo, 
which  is  presented  below  in  its  entirety.  The  memo  describes  actions  taken  to  mitigate  against 
missing  TOI  in  the  stage  2  dig  list. 
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Failure  Analysis  Memo 


Site:  Fort  Ord 

Analyst:  Black  Tusk  Geophysics 
Data:  MM  Cued 
Date:  June  29,  2015 

Our  stage  1  dig  list  missed  two  QC  seeds  (Table  3). 


Table  3.  List  of  Missed  QC  Seeds. 


Anomaly 

Depth  (cm) 

Identification 

Dig  Number 

FO-40248 

26 

Small  ISO 

726 

FO-25074 

30 

Unidentified  large  TOI  (75mm?) 

2171 

a.  Analysis  of  the  factors  that  resulted  in  the  misclassification  of  each  missed  seed 

FO-40248  (small  ISO  at  26  cm  depth) 

During  the  training  portion  of  the  data  analysis,  we  specifically  requested  ground  truth  for 
several  anomalies  with  excellent  matches  to  the  IVS  and  test  pit  small  ISO  polarizabilities 
(Figure  18).  All  of  these  were  identified  as  non-TOI  (frag).  On  this  basis  we  removed  the 
small  ISO  reference  polarizabilities  from  our  reference  library  and  hence  missed  the  QC 
seed. 


Figure  18.  Polarizabilities  for  Seven  Anomalies  with  Excellent  Matches  to  Our  Small 
ISO  Reference  Items  (which  were  based  on  local  IVS  and  test  pit  measurements). 

We  requested  training  data  for  all  of  these;  all  were  identified  as  frag. 
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FO-25074  (75mm?  at  30  cm  depth) 

This  seed  item  was  not  identified  in  the  ground  truth  file,  but  it  appears  to  be  a  75mm 
projectile.  A  re-inspection  of  the  data  for  this  anomaly  (Figure  19)  shows  that  several  of  the 
components  have  relatively  large  misfits.  Ideally,  this  anomaly  should  have  been  classified  as 
“cannot  analyze”.  Its  misfit  metric  was,  however  not  quite  large  enough  for  this  anomaly  to 
be  flagged  for  visual  inspection  for  a  decision  to  be  made  by  the  analyst.  Thus,  essentially 
this  TOI  was  missed  because  the  metric  used  for  flagging  poor  data  fits  was  not  low  enough, 
i.e.,  analyst  error. 


RxX-TxX  RxY-TxX  RxZ-TxX 


RxX-TxY  RxY-TxY  RxZ-TxY  RxX-TxZ  RxY-TxZ  RxZ-TxZ 


Figure  19.  Data  for  Missed  Seed  FO-25074  (75mm). 

Several  components  have  large  misfits  and  thus  this  anomaly  should  have  been  classified  as  “cannot 
analyze  Blue  —  observed  data;  green  —  predicted  data. 

b.  Description  of  how  the  analysis  procedures  have  been  modified  based  on  the 
additional  information  provided 

We  have  re-inserted  the  small  ISO  reference  polarizabilities  into  our  reference  library  to 
ensure  that  all  small  ISOs  with  a  good  match  to  these  items  will  be  dug.  We  have  also  re¬ 
examined  anomalies  with  large  data  misfits  to  looks  for  other  cases  of  anomalies  that  should 
be  classified  as  “cannot  analyze”.  An  additional  19  items  were  classified  as  “cannot  analyze”. 
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c.  Evidence  that  the  modified  analysis  scheme  correctly  classifies  the  missed  seeds  and 
can  reasonably  be  expected  to  correctly  classify  all  remaining  TOI. 

With  the  addition  of  the  small  polarizabilities  into  our  referenee  library,  the  missed  seed  FO- 
40248  would  have  been  dug  because  its  polarizability  misfit  is  0.493,  whereas  we  were 
digging  all  items  with  misfits  less  than  0.6.  Without  the  small  ISO  polarizabilities  in  the 
reference  library,  its  misfit  was  0.624  (with  a  best  match  against  a  37mm  item),  and  hence 
was  not  originally  dug.  Other  items  with  good  matches  to  our  small  ISO  reference  items  will 
now  be  dug. 

We  have  also  re-inspected  items  with  poor  misfits  to  ensure  that  anomalies  with  poor  data 
fits  from  all  inversions  are  classified  as  “cannot  analyze”. 


In  addition  to  the  steps  described  in  the  above  failure  memo,  we  augmented  our  ordnance 
reference  library  with  five  new  items  based  on  the  TOI  found  in  the  stage  I  dig  list.  For  the  stage 
2  dig  list  we  dug  an  additional  245  items.  The  stage  2  partial  ROC  curve  is  shown  in  Figure  20. 
Five  more  TOI  were  found.  Three  of  these  (FO-2502I,  25129  and  50048)  were  anomalies  which, 
after  re-QC,  were  classified  as  cannot  analyze.  Of  the  remaining  two,  one  (FO-40258)  was  a 
small  ISO.  The  other  (FO-25I05)  was  a  75mm  projectile. 


TOI1or2 

FtOrd  BtackTusk  PolMatch  None  MetalMapper  Custom  s2  v1 


Figure  20.  Partial  ROC  Curve  for  the  Stage  2  Dig  List. 
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After  additional  QC,  one  more  anomaly  was  classified  as  cannot  analyze.  Inspection  of  the 
polarizabilities  for  the  anomalies  after  the  stop  dig  point  of  the  stage  2  dig  list  revealed  no 
remaining  high-likelihood  TOI.  As  a  simulated  quality  assurance  (QA)  measure,  we  decided  to 
select  an  additional  25  of  the  most  likely  anomalies  (visually,  based  on  polarizability  shape  and 
size)  for  digging  in  stage  3.  Several  of  these  looked  like  potential  40mm  projectiles,  but  had 
polarizability  misfits  with  respect  to  the  40mm  reference  items  that  were  too  large  to  result  in 
them  being  placed  on  the  dig  list.  The  partial  ROC  curve  for  stage  3  is  shown  in  Figure  21.  No 
additional  TOI  were  found. 


TOI1or2 

FtOrd  BlackTusk  PolMatch  None  MetalMapper  Custom  s3  v1 


Figure  21.  Partial  ROC  Curve  for  the  Stage  3  Dig  List. 


With  no  additional  TOI  found  in  the  stage  3  dig  list,  digging  was  terminated  and  the  stage  3  list 
was  declared  to  be  the  final  dig  list.  Separate  final  scoring  (Figure  22  and  Figure  23)  was 
received  for  the  two  objectives  of  this  study  (finding  all  large  TOI  to  a  depth  of  60  cm,  and 
detecting  all  munitions). 
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TOM 

FtOrd  BlackTusk  PolMatch  None  MetalMapper  Custom  s3  v1 


Figure  22.  Final  Scoring  for  Primary  Objective  (finding  all  large  TOI  to  a 

depth  of  60  cm). 

Orange  dot  denotes  location  of  last  TOI.  Blue  dot  is  stop  dig  point. 


TOI1or2 

FtOrd  BlackTusk  PolMatch  None  MetalMapper  Custom  s3  v1 


Difficult  TOIs: 

#FO-20569  (35mm  legacy) 
#FO-10032  (60mm  legacy) 
#FO-4077 1  (20mm  legacy) 
#FO-20504  (35mm  legacy) 
#FO-40655  (20mm  legacy) 
#FO-40546  (20mm  legacy) 
#FO-10808  (40mm  legacy) 
#FO-20633  (35mm  legacy) 
#FO-20164  (40mm  legacy) 
#FO-10228  (40mm  legacy) 

FO-1738(40mm  legacy) 


Figure  23.  Final  Scoring  for  Secondary  Objective  (detecting  all  TOI). 

Eleven  TOI  occur  after  the  stop  dig  point.  Orange  dot  denotes  location  of  last  TOI.  Blue  dot  is  stop  dig  point. 

The  primary  objective  of  finding  all  large  TOI  to  a  depth  of  60  cm  was  easily  achieved  (Figure 
22).  Because  we  did  not  submit  a  dig  list  specifically  tailored  to  finding  only  large  TOI,  the  stop 
dig  point  on  the  final  ROC  curve  in  Figure  22  is  the  stop  dig  point  that  was  used  for  finding  all 
TOI.  If  the  stop  dig  point  were  to  be  taken  as  the  point  in  the  first  stage  dig  list  where 
classification  changed  from  specifically  looking  for  large  TOI  only  to  looking  for  all  TOI  (dig 
#203),  the  stop  dig  point  would  occur  54  digs  after  the  last  TOI  was  found. 
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The  secondary  objective  of  finding  all  TOI  was  not  achieved,  with  eleven  TOI  missed.  Of  the 
356  unique  TOI  on  site,  we  found  345  (96.9%)  with  874  total  digs  of  unique  targets.  Ignoring  the 
eleven  missed  TOI,  the  false  alarm  rate  was  1.53  non-TOI  digs  per  TOI  dig.  Further  analysis  of 
the  missed  TOI  is  presented  in  the  following  retrospective  analysis. 

4.2.3  MetalMapper  cued  retrospective  analysis 

Our  final  dig  list  missed  eleven  TOI  (Table  4):  four  40mm  projectiles,  three  35mm  subcaliber 
rockets,  three  20mm  projectiles,  and  one  60mm  illumination  round.  We  will  look  at  each  group 
of  similar  target  types  separately. 


Table  4.  List  of  Eleven  Missed  TOI  Ordered  by  Dig  Number. 

All  munitions  except  the  60mm  illumination  round  are  practice  rounds. 


Anomaly 

Depth 

(cm) 

Identification 

Dig 

Number 

FO- 10738 

19 

40mm  M9 18 

989 

FO- 10228 

2 

40mm  M918  (nose  only) 

1167 

FO-20164 

3 

40mm  M781  +  frag 

1251 

FO-20633 

10 

35mm  subcaliber  rocket  M73 

1318 

FO- 10808 

1 

40mm  M918  (nose  only)  +  frag 

1322 

FO-40546 

3 

20mm  M99 

1688 

FO-40655 

6 

20mm  M99 

1886 

FO-20504 

8 

35mm  subcaliber  rocket  M73 

2045 

FO-40771 

12 

20mm  M99 

2384 

FO- 10032 

23 

60mm  illumination  round  M83 

2459 

FO-20569 

12 

35mm  subcaliber  rocket  M73 

2552 

4.2.3.1  Missed  TOI:  40mm  (4:  FO-10738, 10228,  20164  and  10808) 

The  four  missed  40mm  projectiles  are  shown  in  Figure  24. 
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Figure  24.  Four  Missed  40mm  Projectiles. 
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Missed  TOI  FO-10738 


As  shown  in  Figure  25,  the  location  of  the  excavated  40mm  projectile  is  about  1.4  m  east  of  the 
target  pick  and  sensor  location.  In  addition  to  the  large  offset,  this  item  was  located  at  a  depth  of  19 
cm.  Of  the  89  40mm  projectiles  recovered  during  this  project,  this  was  the  greatest  depth  found. 
Given  the  very  large  offset,  relatively  deep  depth,  and  small  size  of  the  target,  it  is  not  surprising 
that  this  TOI  was  missed.  Furthermore,  as  shown  in  Figure  25,  our  results  suggest  that,  with  very 
little  doubt,  a  50  caliber  projectile  is  located  at  the  target  pick  location  of  FO- 10738,  yet  the  ground 
truth  information  makes  no  mention  of  this.  Our  conclusion  for  this  anomaly  is  that  (a)  the  40mm 
projectile  should  not  be  associated  with  this  anomaly;  and  (b)  the  ground  truth  information  is 
suspect  because  it  did  not  report  finding  a  50  caliber  projectile  at  the  target  pick  location. 


Figure  25.  Left:  Sensor  and  Ground  Truth  location  for  Missed  TOI  FO-10738  (40mm). 
Right:  Polarizabilities  for  the  Single  Object  Inversion  Show  that  a  50  Caliber  Projectile  is 
Almost  Certainly  Located  at  the  Target  Pick  Location  (at  a  depth  of  ~3  cm),  Yet  the 
Ground  Truth  Information  Has  No  Mention  of  This. 

Left:  Labeled  yellow  square  is  target  pick  location  for  FO-10738.  Light  grey  lines  and  dots  show  location 
of  MetalMapper  transmitters  and  receivers  (short  grey  line  point  south  shows  that  front  face  of  the  sensor 
is  facing  south).  Numbered  white  and  yellow  circles  are  predicted  model  locations  from  single  and  multi¬ 
object  inversions.  Broken  black  line  shows  extent  of  inversion  bounds  (predicted  models  are  constrained 
to  lie  within  these  bounds).  Labeled  magenta  dot  is  location  of  a  recollect.  Black  x  shows  ground  truth 
location  of  the  40mm  projectile,  which  is  approximately  1.4  m  east  of  the  target/sensor  center. 

Missed  TOIs  FO-I0228  and  10808 

Both  of  these  are  nose  parts  only  of  a  40mm  projectile.  We  had  encountered  a  40mm  nose  part  in 
our  stage  1  dig  list  and  the  polarizabilities  from  this  item  (FO- 10372)  were  added  to  our 
reference  library.  The  polarizabilities  for  FO- 10229  and  10808  (Figure  26)  show  a  distinct 
resemblance  to  the  40mm  nose  reference  polarizabilities,  however,  the  misfit  values  (0.711  and 
1.171,  respectively)  were  larger  than  our  stop  dig  threshold  (0.6)  and  so  these  items  appeared 
after  the  stop  dig  point.  The  40mm  nose  part  is  the  smallest  item  in  our  reference  library,  and  in 
retrospect  it  is  clear  that  there  is  significant  variation  in  the  polarizabilities  for  this  item. 
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Figure  26.  Polarizabilities  for  Missed  40mm  Projectiles  FO-10228  (left)  and  FL-10808 

(right). 

Both  are  nose  parts  only,  and  both  show  similarities  with  our  40mm  nose  reference  polarizabilities 
(broken  grey  lines);  however,  the  misfit  values  (yellow  labels)  were  large  enough  that  these  items 

appeared  after  the  stop  dig  point. 


Missed  TOI  FO-20164 

The  40mm  projectile  at  anomaly  FO-20164  can  be  considered  a  “one-off’:  it  is  an  M781  type 
practice  round,  while  all  other  40mm  projectiles  classified  as  TOI  for  this  study  are  M918  type. 
The  88  M918  TOI  were  all  found  in  Grids  12A  and  12B,  while  a  total  of  19  M781  types  were 
found,  all  in  grid  11-B;  however  only  one  of  these  (FO-20164)  is  classified  as  a  TOI  in  the 
ground  truth.  In  fact  we  did  receive  training  data  for  an  M78I  type  40mm:  FO-20446,  however 
(a)  it  was  classified  as  MD,  and  (b)  its  polarizabilities  were  distinctly  different  from  those  of 
FO-20164.  The  polarizabilities  for  FO-20446  are  essentially  identical  to  the  40mm  nose 
polarizabilities  from  our  reference  library. 

The  M781  is  smaller  than  the  M918.  We  also  note  that  the  projectile  was  actually  located  about 
1.3  m  SW  from  the  target  pick  for  FO-20164.  FO-20164  was  acquired  a  total  of  6  times.  The 
sixth  acquisition  was  only  about  25  cm  from  the  projectile,  so  that  the  projectile  was  well  within 
the  footprint  of  the  sensor.  The  polarizabilities  for  FO-20164  (Figure  27)  are  distinct  from  all  the 
40mm  polarizabilities  in  our  reference  library.  This  is  a  classic  example  demonstrating  the 
difficulty  in  finding  distinct,  small  one-off  items. 
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T20164-C837-Mod  4  (Inv  3/3=301:  1/3)  [P] 


Figure  27.  Polarizabilities  for  Missed  40mm  Projectiles  FO-20164. 

Broken  grey  lines  are  for  a  reference  40mm  measurement  made  at  the  test  pit.  This  item  is  a  one-off 
M781;  all  other  40mm  TOI found  during  this  study  are  the  larger  M91 8  type. 


4.2.3. 2  Missed  TOI:  35mm  subcaliber  rockets  (3:  FO-20633,  20504  and  20569) 

The  three  missed  35mm  subcaliber  rockets  are  shown  in  Figure  28.  While  these  three  items  look 
identical,  the  recovered  polarizabilities  vary  significantly,  hence  there  was  no  chance  of 
identifying  these  as  belonging  to  a  small  cluster.  For  comparison  purposes  we  will  use  the 
polarizabilities  from  FO-20504  as  a  reference  for  the  35mm  rocket.  The  rocket  at  this  anomaly  is 
the  most  shallow  of  the  three,  and  the  only  one  where  a  piece  of  scrap  was  not  found  along  with 
the  TOI.  But  even  for  FO-20504,  it  is  not  clear  which,  if  any  of  the  models,  are  truly 
representative  of  the  35mm  rocket.  We  use  one  of  the  models  from  the  2-object  inversion  for  the 
following  retrospective  analysis. 
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Figure  28.  Three  Missed  35mm  Subcaliber  Rockets. 


29 


Missed  TOI  FO-20633 


A  large  conical  fuze,  resembling  an  M54  fuze  (for  example  found  at  Camp  Beale),  was  found  at 
this  location  in  addition  to  the  35mm  rocket.  The  fuze  was  found  a  more  shallow  depth  (8  versus 
10  cm)  and  is  closer  to  the  center  of  the  sensor  (3  versus  27  cm).  Recovered  polarizabilities  for 
this  anomaly  (Figure  29)  show  that  the  reference  item  with  the  best  fit  is  an  M54  fuze.  The 
predicted  location  and  depth  for  these  models  are  consistent  with  the  fuze  that  was  found.  None 
of  the  other  models  resembles  the  35mm  reference  polarizabilities  taken  from  FO-20504.  Note 
that  the  M54  reference  item  shown  in  Figure  29  was  not  used  in  the  reference  ordnance  library 
used  for  classification  because  fuzes  were  not  considered  TOI. 


T20633-C1471-Mod  1  (Inv  1/3=SOI:  1/1)  [F]  T20633-C1471-Mod  2  (Inv  2/3=201:  1/2)  [P]  T20633-C1 471 -Mod  3  (Inv  2/3=201:  2/2)  [P] 


T20633-C1471-Mod  4  (Inv  3/3=301:  1/3)  [P]  T20633-C1471-Mod  5  (Inv  3/3=301:  2/3)  [F]  T20633-C1471-Mod  6  (Inv  3/3=301:  3/3)  [P] 


Figure  29.  Polarizabilities  for  All  Six  Models  of  FO-20633  (missed  35mm  subcaliber 
rocket)  Along  with  the  Best  Fitting  Polarizabilities  from  the  Reference  Library  (broken 

grey  lines). 

Note  that  one  of  the  models  (lower  left)  is  a  close  match  to  an  M54  fuze.  A  fuze  that  looks  like  an  M54 
was  also  found  at  this  anomaly  location.  None  of  the  polarizabilities  match  those  from  FO-20504.  Top 
row:  models  for  1-  and  2-object  inversions;  bottom  row:  models  for  3-object  inversion.  Numbers  in  lower 
left  are  misfit  values  between  observed  and  reference  polarizabilities  (smaller  number  implies  better  fit). 


Missed  TOI  FO-20504 

Of  the  three  missed  35mm  rockets,  this  was  the  most  shallow  (8  cm),  closest  to  the  center  of  the 
sensor  (3  cm  offset),  and  the  only  one  not  accompanied  by  a  piece  of  scrap.  In  theory  this  item 
should  produce  the  most  representative  set  of  polarizabilities  for  the  35mm  rocket,  hence  we  are 
using  one  of  the  models  from  this  anomaly  as  the  de-facto  reference  for  the  35mm  rocket  in  this 
retrospective  analysis.  Typically  when  a  model  can  be  well  constrained,  the  polarizabilities  for  the 
single  object  inversion,  and  the  polarizabilities  from  one  model  each  of  the  2-  and  3-object 
inversions  should  all  look  similar.  In  this  case  that  is  not  true  as  can  be  judged  by  the  large  misfit 
values  for  all  but  the  second  model  in  Figure  30.  The  reference  polarizabilities  based  on  FO-20504 
are  actually  fairly  typical  of  small  scrap,  hence  the  difficulty  in  finding  this  TOI  (e.g..  Figure  31). 
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Figure  30.  Polarizabilities  for  All  Six  Models  of  FO-20504  (missed  35mm 

subcaliber  rocket). 

The  reference  polarizabilities  (broken  grey  lines)  are  based  on  the  second  model  (top  row,  center).  Note 
the  inconsistency  of  the  models.  Top  row:  models  for  1-  and  2-object  inversions;  bottom  row:  models  for 
3-object  inversion.  Numbers  in  lower  left  are  misfit  values  between  observed  and  reference 
polarizabilities  (smaller  number  implies  better  fit). 


T20484-C1 265-Mod  2  (Inv  2/3=201:  1/2)  [P] 


FO_20484_ORD_11B_CUED_121714_00197_20484.csv 


Figure  31.  Polarizabilities  for  FO-20484  (small  frag)  Plotted  Against  the  Polarizabilities 
for  FO-20504  (missed  35mm  rocket  TOI;  broken  grey  line). 

This  illustrates  how  the  35mm  rocket  is  basically  indistinguishable  from  small  frag. 
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Missed  TOI  FO-20569 


This  35mm  rocket  was  challenging  because  it  was  located  outside  the  footprint  of  the  sensor 
(Figure  32),  at  an  offset  of  59  cm  from  the  center  of  the  sensor.  It  is  the  deepest  of  the  three 
missed  35mm  rockets  and  there  is  a  small  piece  of  frag  closer  to  the  center  of  the  sensor  (12  cm 
offset)  at  a  depth  of  only  2  cm.  One  of  the  3-object  inversion  models  (model  6)  is  located  near 
the  position  of  the  rocket,  but  the  polarizabilities  for  this  model  (Figure  33)  are  very  dissimilar  to 
the  reference  polarizabilities  based  on  FO-20504.  None  of  the  other  models  are  a  good  match, 
either.  It  is  most  likely  that  models  2  and  5  represent  the  small  frag.  It  seems  that,  because  of  its 
position,  depth  and  relatively  small  size,  the  rocket  is  not  resolved  by  the  data. 


Figure  32.  Ground  Truth  Location  of  the  35nini  Rocket  at  FO-20569  (“X”)  versus 
Sensor  Location  (light  grey  lines  and  dots  show  MetalMapper  transmitters  and  receivers). 

Note  that  the  rocket  lies  outside  the  footprint  of  the  sensor  (59  cm  from  center  of  sensor).  Numbered 
circles  are  the  predicted  model  locations  for  the  three  inversions.  Model  6  (just  below  the  X)  is  the  model 
nearest  to  the  rocket,  but  its  polarizabilities  are  a  very  poor  match  to  the  reference  polarizabilities  for  a 

35mm  rocket. 
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T20569-C1 379-Mod  1  (Inv1/3=S0I:  1/1)  [F]  T20569-C1 379-Mod  2  (Inv  2/3=201:  1/2)  [P]  T20569-C1 379-Mod  3  (Inv  2/3=201:  2/2)  [F] 


T20569-C1 379-Mod  4  (Inv  3/3=301:  1/3)  [F]  T20569-C1 379-Mod  5  (Inv  3/3=301:  2/3)  [P]  T20569-C1 379-Mod  6  (Inv  3/3=301:  3/3)  [F] 


Figure  33.  Polarizabilities  for  All  Six  Models  of  FO-20569  (missed  35mm  subcaliber 
rocket)  Plotted  against  Reference  35mm  Polarizabilities  Based  on  FO-20504. 

Note  the  inconsistency  of  the  models.  Top  row:  models  for  1-  and  2-object  inversions;  bottom  row: 
models  for  3-object  inversion.  Numbers  in  lower  left  are  misfit  values  between  observed  and  reference 

polarizabilities  (smaller  number  implies  better  fit). 


4.2.3.3  Missed  TOI:  20mm  projectiles  (3:  FO-40546,  40655  and  40771) 

The  three  missed  20mm  projectiles  are  shown  in  Figure  34,  and  their  polarizabilities  are  shown 
in  Figure  35.  In  principle,  these  three  missed  TOI  represent  the  most  serious  miss  for  a  number 
of  reasons:  The  polarizabilities  for  these  items  are  quite  consistent  and  therefore  one  or  more  of 
these  items  could  have  been  found  through  a  more  thorough  cluster  analysis.  Also,  the 
polarizabilities  are  fairly  consistent  with  20mm  reference  items  in  our  collection  of  reference 
items.  In  practice,  due  to  their  small  size,  and  because  a  lot  of  small  scrap  has  very  similar 
polarizabilities,  it  is  still  difficult  to  find  20mm  items  without  digging  a  huge  amount  of  scrap.  A 
cluster  that  would  include  the  three  missed  TOI  would  likely  include  a  lot  of  other  items,  so  that, 
short  of  being  lucky  enough  to  choose  one  of  the  TOI  for  training  out  of  the  large  cluster,  the 
only  way  to  assure  digging  all  20mm  TOI  would  be  to  dig  a  very  large  amount  of  small  scrap. 
We  did,  in  fact,  request  training  data  for  some  20mm-like  items  (FO-30345  and  20401),  but  both 
of  these  turned  out  to  be  small  frag.  Fortunately  the  objective  for  this  study  was  to  find  95%  of 
all  TOI,  so  that,  even  with  the  eleven  missed  TOI,  we  did  achieve  that  goal. 
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Figure  34.  Three  Missed  20mm  Projectiles:  FO-40546,  40655  and  40771. 


Polarizabilities  for  3  models 


0.5  1  5  10 


Figure  35.  (1-3)  Polarizabilities  for  the  Three  Missed  20mm  Projectiles. 

Reference  polarizabilities  (broken  grey  lines  are  taken  from  the  UX-Analyze  library  for  an  M55  20mm 
projectiles  at  10  cm  depth.  Numbers  in  yellow  are  misfits  with  respect  to  the  reference  polarizabilities. 
Note  that  for  FO-4771,  the  misfit  value  was  large  enough  so  that,  given  our  dig  threshold  of  0.6,  this  item 
would  not  have  been  dug  even  if  we  were  looking  for  20mm  projectiles.  Right:  polarizabilities  overlain  to 

show  consistency. 


4.2.3. 4  Missed  TOI:  60mm  illumination  round  (F 0-10032) 

The  polarizabilities  for  FO- 10032  are  shown  in  Figure  36.  None  are  a  good  match  to  Ford  Ord 
test  pit  measurements  of  a  60mm  item.  Models  1,  3  and  5  are  self-consistent,  suggesting  these 
are  the  representative  response  for  this  item.  However,  these  polarizabilities  are  not  a  good  match 
to  any  of  the  33  60mm  items  in  our  large  reference  library. 
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T10032-C28-S0^1-P 


T10032-C28-20I/1-F 


T10032-C28-2OI/2-P 


Figure  36.  Left:  Ground  Truth  Photo  for  Missed  60mm  FO-10032.  Right: 
Polarizabilities  for  All  Models  Plotted  Against  60mm  Polarizabilities  Taken  from  Test  Pit 

Measurements  at  Fort  Ord  (broken  grey  lines). 


Surprisingly,  two  of  the  models  (2  and  4)  are  actually  fairly  consistent  with  measurements  of  a 
60mm  item  made  at  Camp  Beale  (Figure  37).  However,  the  location  of  both  of  these  models  lies 
to  the  southeast  on  the  horizontal  inversion  boundary,  at  a  distance  of  80-85  cm  from  the  center 
of  the  sensor  (which  is  located  precisely  on  the  target  pick).  Both  models  are  actually  much 
closer  to  the  target  location  for  FO-10117.  This  anomaly  corresponds  to  a  shallow  (1  cm)  40mm 
projectile,  so  the  similarity  between  the  polarizabilities  of  model  2  and  4  with  the  Beale  60mm 
polarizabilities  may  be  nothing  more  than  coincidence.  Both  models  2  and  4  were  failed  (because 
of  their  large  offset  from  the  sensor  location  and  apparent  association  with  target  FO-101 17),  and 
hence  not  used  for  classification.  Regardless,  because  the  Beale  60mm  polarizabilities  were  not 
included  in  library  used  for  classification  at  Fort  Ord,  this  TOI  was  destined  to  be  not  dug. 
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T10032-C28-Mod  2  (Inv  2/3=201:  1/2)  [F] 


Figure  37.  Left:  Predicted  Model  Locations  for  FO-10032  (numbered  circles).  Right: 

Polarizabilities  for  Model  2. 


Left:  Light  grey  lines  and  dots  show  the  MetalMapper  receiver  and  transmitter  locations.  Broken  black 
line  are  the  horizontal  inversion  bounds.  Models  2  and  4  lie  on  the  inversion  bounds  near  the  target 
location  ofFO-10117.  Thick  “X”  is  the  reported  location  of  the  60mm  round.  Right:  These  are  a  fairly 
good  match  to  a  60mm  item  from  Camp  Beale.  Both  model  2  and  4  were  considered  unreliable  due  to 
their  large  offset  from  the  center  of  the  sensor  (80-85  cm)  and  their  apparent  association  with  FO-10117. 
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5.0  CONCLUSIONS  AND  RECOMMENDATIONS  FOR  FUTURE 
WORK 

A  key  result  of  this  demonstration  is  the  reduction  in  false  alarms  that  can  be  achieved  using  long 
off-time  (25  ms)  dynamic  MetalMapper  data.  If  the  primary  objective  of  detecting  large  TOI  is 
pursued  in  future  work,  then  the  MetalMapper  should  be  used  for  DGM.  Although  dynamic 
MetalMapper  production  rates  will  be  significantly  lower  than  with  a  conventional  EM61  array, 
the  MetalMapper’ s  improved  resolution  and  clutter  rejection  are  necessary  for  reliable  detection 
of  large  TOI  at  Fort  Ord. 

For  this  demonstration,  the  primary  objective  of  confident  classification  of  large  munitions  such 
as  155mm  projectiles  was  achieved  using  the  MetalMapper  cued  data  acquired  in  low,  medium 
and  high  density  grids.  All  large  TOI  were  identified  in  our  classification  diglist.  However,  two 
missed  seed  items  in  our  stage  one  submission  do  indicate  that  a  conservative  approach  is 
required  to  achieve  the  primary  objective.  In  particular,  we  found  that  the  data  misfit  threshold 
for  identifying  “can’t  analyze”  scenarios  had  to  be  relaxed  in  order  to  ensure  that  all  large  TOI 
were  identified. 

There  is  a  significant  range  of  TOI  at  Fort  Ord,  and  we  found  that  some  smaller  items  (e.g.  35 
mm  rockets)  were  difficult  to  distinguish  from  the  clutter  at  this  site.  Although  there  were  11 
small  TOI  missed  in  our  cued  MetalMapper  diglist,  the  classification  results  indicate  that  95%  of 
smaller  munitions  can  be  confidently  classified  within  the  range  of  background  conditions  at  Fort 
Ord. 
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